Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning

Visual storytelling is a task of creating short story based on photo streams. Different from visual captions, stories contain not only factual descriptions, but also imaginary concepts that do appear in the images. In this paper, we propose novel imagine-reason-write generation framework (IRW) for storytelling, inspired by logic humans when they write story. First, an imagine module leveraged to learn imaginative storyline explicitly, improving coherence and reasonability generated Second, employ reason fully exploit external knowledge (commonsense base) task-specific (scene graph event graph) with relational reasoning method storyline. way, can effectively capture most informative commonsense relationships among objects images, which enhances diversity informativeness Finally, integrate generate human-like original semantics Extensive experiments benchmark dataset (i.e., VIST) demonstrate proposed IRW significantly outperforms state-of-the-art methods across multiple evaluation metrics.